interesting! we could probably parse it on HF side in the future if it makes sense and if it unlocks cool features (we already attempt to extract quantization type from filename but this could make it more robust. cc @mishig25)

If it helps, we follow somewhat similar (but not exhaustive) in the gguf-my-repo quantisation space: https://huggingface.co/spaces/ggml-org/gguf-my-repo/blob/main/app.py#L67

Standardisation in file name is always a great move!

If it helps, we follow somewhat similar (but not exhaustive) in the gguf-my-repo quantisation space: https://huggingface.co/spaces/ggml-org/gguf-my-repo/blob/main/app.py#L67

Standardisation in file name is always a great move!

Had a quick look. Do you mean your current naming arrangement is kind of like

• <model_name>/<model_name.lower()>.<q_method.upper()>.gguf so something like TinyStories/tinystories-F16.gguf

If so then do you have a preferred form? I came to this form basically by casual observation of typical naming scheme in huggingface, hence <Model>-<Version>-<ExpertsCount>x<Parameters>-<Quantization>.gguf (where version field is optional).

But obviously it's research by vibes, so it be better if I had some feedback, especially for those that be forced to try and parse such files. Ergo @JidongZhang-THU would it make it easier for you if we made 'version' not optional? (Expert count being optional is okay as it's easy to tell that x is either there or not there).

And of course... did I miss anything that would be useful for people parsing models file names?

Let me add a bit of backstory as to why we chose this naming scheme (which I'm more than happy to change):

A typical user of the quantisation space would want to create quants for an arbitrary model on the Hub.
To preserve the info about the model itself, we copy the model-id (which can be different from the model name, for example there are 4.5k+ mixtral models on the hub all with different names based on their training set, and other metadata.) and then append the information about the quant time along with.

The model name typically already would have the information about expert count + parameters.

I'm open to ideas to align better, I just thought I'd provide more context.

@julien-c do you have a preference when it comes to parsing filenames? I'm basically treating it as a sort of - dash separated value (in which case, I should probably make the version field mandatory).

@mofosyne no, we'll adapt!

Thanks for the historical context. I might have gotten a bit crazy here, but I've ended up mapping each enum name to the tensor type description and the historical context behind each PR that relates to it's initial inclusion...

Not even sure if it's allowed on this gguf.md page, so just attaching it to this comment in case I should remove it. But hopefully it helps provide a general glance of each gguf file type.

Oh and i've updated the page a bit. Opted for 'tensor type' rather than 'file name' as that appears to make more sense to me at least.

@mofosyne, I've made a similar table of quant descriptions in https://huggingface.co/docs/hub/gguf#quantization-types (sharing just in case if there's any useful info)

@mishig25 thanks. Decided to cross reference your table with what I got and this is the breakdown i was able to figure out. I'm not 100% sure on all the superblock configuration for the i-quantization based on your statement and the llama.cpp PR description, but I was able to extract some out. I think gg would be a clearer source of truth here (especially some of my general assertions below).

Encoding Scheme Name Table

• All superblocks have fp16 scaling factor and contains up to 256 weights. Number of weights in a block must be divisible by 256.

@mishig25 when you made the table, were you able to figure out the superblocks makeup and how to represent the weight formulae (in general)? Also in your opinion, is this table in the right location or should it be split up (and if so then where)?

(And on a meta note... how much information should we really expose in this document... too much can confuse developers)

edit: Justine T also mentioned regarding my 'Weights Encoding Scheme' table is that I may have issue using different name for quants than what the software (presumably llama.cpp) uses. So I guess we could say this is not a super hard and fast mapping, but can include other variants... but for the context of ggml this is the base scheme name. llama.cpp can then define their own extra naming (e.g. _S, _M and _L) in their own documentation (as extra pointers for users of what to expect).

(And on a meta note... how much information should we really expose in this document... too much can confuse developers)

IMO, the entire encoding section should just be reduced to simply:

Indicates the weights encoding scheme that was applied to the model. Content, type mixture and arrangement however are determined by user code and can vary depending on project needs.

The rest of the information is specific mainly to llama.cpp and not relevant to the GGUF format

@ggerganov thanks, looks much more compact and focused now.

@mishig25 @julien-c @Vaibhavs10 @Green-Sky let's lock this in?

(Wonder if the table with bits, datatype, block config, etc... be useful anywhere, such as llama.cpp documentation and if so then which specific location)

@ggerganov thanks for the merge

I've decided to place the table to https://github.com/ggerganov/llama.cpp/wiki/Tensor-Encoding-Schemes . Turns out github wiki kind of suck at rendering tables, but hope it's of help to everyone here.